<html>
<head>
  <title>小例子</title>
  <basefont face="微软雅黑" size="2" />
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
  <meta name="exporter-version" content="Evernote Windows/307027 (zh-CN, DDL); Windows/6.1.0 (Win32);"/>
  <style>
    body, td {
      font-family: 微软雅黑;
      font-size: 10pt;
    }
  </style>
</head>
<body>
<a name="502"/>
<h1>小例子</h1>

<div>
<span><div><b style="color: rgb(28, 51, 135); font-size: 16pt; line-height: 1.45;">1、预设值：模块和参数设置</b><br/></div><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">import numpy as np</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">import pandas as pd</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">import time</span></div><div><br style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"/></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">N_STATES = 6 # 1维世界的宽度</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">ACTIONS = ['left', 'right'] # 探索者的可用动作</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">EPSILON = 0.9 # 贪婪度 greedy</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">ALPHA = 0.1 # 学习率</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">GAMMA = 0.9 # 奖励递减值</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">MAX_EPISODES = 13 # 最大回合数</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">FRESH_TIME = 0.3 # 移动间隔时间</span></div></div><div><br/></div><div><span style="font-weight: bold; color: rgb(28, 51, 135); font-size: 16pt; line-height: 1.45;">2、Q表初始化：将所有的Q values（行为值）放在q_table中<span style="font-weight: bold; font-size: 16pt; color: rgb(28, 51, 135);">, 更新 q_table 也是在更新他的行为准则. q_table 的 index 是所有对应的 state (探索者位置), columns 是对应的 action (探索者行为).</span></span></div><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-weight: bold; color: rgb(28, 51, 135); font-size: 16pt; line-height: 1.45;"><span style="font-weight: bold; font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">def build_q_table(n_states,actions):</span></span></div><div><span style="font-weight: bold; font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    table = pd.DataFrame(</span></div><div><span style="font-weight: bold; font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    np.zeros((n_states,len(actions))),</span></div><div><span style="font-weight: bold; font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    columns = actions,</span></div><div><span style="font-weight: bold; font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    )</span></div><div><span style="font-weight: bold; font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    return</span></div></div><div><br/></div><div><span style="font-weight: bold; color: rgb(28, 51, 135); font-size: 16pt; line-height: 1.45;"><br/></span></div><div><span style="font-weight: bold; color: rgb(28, 51, 135); font-size: 16pt; line-height: 1.45;">3、定义动作：</span></div><div><ul><li>epsilon greedy 的概念，因为在初始阶段, 随机的探索环境, 往往比固定的行为模式要好, 所以这也是累积经验的阶段, 我们希望探索者不会那么贪婪(greedy). 所以 EPSILON 就是用来控制贪婪程度的值. EPSILON 可以随着探索时间不断提升(越来越贪婪), 不过在这个例子中, 我们就固定成 EPSILON = 0.9, 90% 的时间是选择最优策略, 10% 的时间来探索.</li></ul></div><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">def choose_action(state,q_table):</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    state_actions = q_table.iloc[state,:]</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    if(np.random.uniform()&gt;EPSILON) or (state_actions.all()==0):</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        action_name = np.random.choice(ACTIONS)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    else:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        action_name = state_actions.argmax()</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    return action_name</span></div></div><div><br/></div><div><font color="#1C3387"><span style="font-size: 21px; line-height: 30px;"><b><br/></b></span></font></div><div><font color="#1C3387"><span style="font-size: 21px; line-height: 30px;"><b>4、<span style="font-weight: bold; font-size: 16pt; color: rgb(28, 51, 135);">环境反馈 S_, R</span></b></span></font></div><div><ul><li>做出行为后, 环境也要给我们的行为一个反馈, 反馈出下个 state (S_) 和 在上个 state (S) 做出 action (A) 所得到的 reward (R). 这里定义的规则就是, 只有当 o 移动到了 T, 探索者才会得到唯一的一个奖励, 奖励值 R=1, 其他情况都没有奖励.</li></ul><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">def get_env_feedback(S,A):</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    if A == 'right':</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        if S == N_STATES - 2:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">            S_ = 'terminal'</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">            R = 1</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        else:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">            S_ = S + 1</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">            R = 0</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    else:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        R = 0</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        if S == 0:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">            S_ = S</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        else:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">            S_ = S - 1</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    return S_,R</span></div></div></div><div><br/></div><div><br/></div><div><font style="font-size: 16pt; color: rgb(28, 51, 135);"><b>5、环境更新</b></font></div><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">def update_env(S, episode, step_counter):</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    # This is how environment be updated</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    env_list = ['-']*(N_STATES-1) + ['T']   # '---------T' our environment</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    if S == 'terminal':</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        interaction = 'Episode %s: total_steps = %s' % (episode+1, step_counter)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        print('\r{}'.format(interaction), end='')</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        time.sleep(2)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        print('\r                                ', end='')</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">    else:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        env_list[S] = 'o'</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        interaction = ''.join(env_list)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        print('\r{}'.format(interaction), end='')</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">        time.sleep(FRESH_TIME)</span></div><div><br style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"/></div></div><div><br/></div><div><br/></div><div><font style="font-size: 16pt; color: rgb(28, 51, 135);"><b>6、强化学习主循环</b></font></div><div>最重要的地方就在这里. 你定义的 RL 方法都在这里体现. 在之后的教程中, 我们会更加详细得讲解 RL 中的各种方法, 下面的内容, 大家大概看看就行, 这节内容不用仔细研究.</div><div><img src="https://morvanzhou.github.io/static/results/reinforcement-learning/2-1-1.png"></img></div><div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902); background-position: initial initial; background-repeat: initial initial;-en-codeblock:true;"><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);">def rl():</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    </span>q_table = build_q_table(N_STATES, ACTIONS) # 初始 q table</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    </span>for episode in range(MAX_EPISODES): # 回合</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    </span></span>step_counter = 0</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    </span></span>S = 0 # 回合初始位置</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    </span></span>is_terminated = False # 是否回合结束</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    </span></span>update_env(S, episode, step_counter) # 环境更新</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    </span></span>while not is_terminated:</span></div><div><br style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"/></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>A = choose_action(S, q_table) # 选行为</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>S_, R = get_env_feedback(S, A) # 实施行为并得到环境的反馈</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>q_predict = q_table.loc[S, A] # 估算的(状态-行为)值</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>if S_ != 'terminal':</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    <span>    </span></span></span></span>q_target = R + GAMMA * q_table.iloc[S_, :].max() # 实际的(状态-行为)值 (回合没结束)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>else:</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    <span>    </span></span></span></span>q_target = R # 实际的(状态-行为)值 (回合结束)</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    <span>    </span></span></span></span>is_terminated = True # terminate this episode</span></div><div><br style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"/></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>q_table.loc[S, A] += ALPHA * (q_target - q_predict) # q_table 更新</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>S = S_ # 探索者移动到下一个 state</span></div><div><br style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"/></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>update_env(S, episode, step_counter+1) # 环境更新</span></div><div><br style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"/></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    <span>    <span>    </span></span></span>step_counter += 1</span></div><div><span style="font-family: Monaco; font-size: 9pt; color: rgb(51, 51, 51);"><span>    </span>return q_table</span></div></div><div><br/></div><div><br/></div></span>
</div></body></html> 